Field of the Invention
The present invention relates to a technique for presenting a composite image of a sensed image and virtual space image to the user.
Description of the Related Art
In recent years, as a technique for seamlessly blending physical and virtual worlds in real time, a so-called MR (Mixed Reality) technique is known. As one MR technique, the following technique is known. That is, an image of an object which nearly matches an object observed from the pupil position of an observer is sensed using a video camera or the like equipped on a video see-through HMD (Head Mounted Display). An image obtained by superimposing a CG (Computer Graphics) on that sensed image is presented to the observer.
The video see-through HMD acquires digital image data of an object by sensing an image of that object using a charge coupled device such as a CCD, and displays an MR image (mixed reality image) obtained by superimposing a CG image on this digital image data. After that, this MR image is presented to the user via a display device such as a liquid crystal element.
A CG image is rendered and is superimposed onto a sensed image based on position and orientation information of an image sensing device such as a video camera equipped on the HMD, that is, the position and orientation information of the head of the HMD user. As typical methods of generating the position and orientation information, the following two methods are available.
In the first method, the position and orientation information is acquired and generated by a three-dimensional position and orientation sensor such as a magnetic sensor or gyro sensor (acceleration, angular velocity). The magnetic sensor is vulnerable to ambient electromagnetic waves, while the gyro sensor is susceptible to a drift phenomenon in which an output voltage drifts. In this way, it is difficult for only the three-dimensional position and orientation sensor to generate stable position and orientation information.
In the second method, information of an index arranged on a physical space is acquired, and position and orientation information is generated based on the acquired information. As the index, a marker having a specific shape or an infrared LED may often be used. The information of that index is acquired using a camera or infrared camera. The position and orientation information is generated based on the size and position of the acquired index and further based on the shape in case of the marker.
A method of locating a camera used to acquire index information is roughly classified into the following two methods.
In the first method, an arrangement required to acquire index information is laid out in an HMD. This method includes a method of attaching an image sensing device to the HMD and acquiring information of the index using this image sensing device. A method in which an infrared camera is arranged near the image sensing device required to sense a physical space image used to generate an MR image, and an infrared marker laid out on a physical space is sensed using that infrared camera to generate position and orientation information is disclosed in Frank Sauer, Fabian Wenzel, Sebastian Vogt, Yiyang Tao, Yakup Genc, Ali Bani-Hashemi, Siemens Corporate Research, Augmented Workspace: designing an AR testbed, Proceedings of the IEEE and ACM International Symposium on Augmented Reality 2000 pp. 47-53.
In the second method, an image sensing device used to acquire index information is arranged at a position where a bird's-eye view around the HMD user can be obtained. With this method, index information can be acquired from a region broader than a case in which the image sensing device is arranged on the HMD depending on the number and layout of image sensing devices.
Japanese Patent Laid-Open No. 2005-351886 discloses a method of generating position and orientation information using infrared marker information attached to an HMD acquired by an infrared camera that can obtain a surrounding bird's-eye view in addition to marker information acquired by an image sensing device of the HMD.
Upon stably acquiring position and orientation information based on information from a broader space, a larger number of indices have to be set on the physical space. Also, in a video see-through HMD that has to use the same field angle since it is technically difficult to set the field angle of an optical system of the HMD to be broader than that of an image sensing system, the field angle of the image sensing system is constrained to that of the display system. When the display system and image sensing system have the same field angle, a CG is displayed in a moment when an index appears in a sensed image, and the continuity of a CG display may often be lost. When index information in a region within a range broader than a displayable range of the display system of the HMD can be obtained, the number of indices to be set can be reduced. However, arranging an image sensing device independently of the image sensing device of the HMD leads to an increase in scale and complexity of the system arrangement.